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TITLE OF THE I WENT I ON 
DISPLAY APPARATUS, IMAGE PROCESSING APPARATUS AND IMAGE 
PROCESSING METHOD, IMAGING APPARATUS, AND PROGRAM 

BACKGROUND OF THE INVENTION 
Field of the Invention 

The present invention relates to display apparatuses, 
image processing apparatuses as well as image processing 
methods, imaging apparatuses, and programs. Particularly, 
the present invention relates to a display apparatus, an 
image processing apparatus, an image processing method, an 
imaging apparatus, and a program that allow an image to be 
presented to a user with a greater sense of presence than 
conventional devices and methods. 

Discussion of the Background 

As an example of a conventional apparatus for 
presenting a user with an image with a sense of presence, a 
display apparatus including an immersion screen is known. 
In the display apparatus including the immersion screen, the 
user is presented with an image that would be perceived by 
the vision of the user assuming that the user were in a 
virtual space (hereinafter referred to as an immersion image 
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where appropriate) . Thus, by viewing the immersion image, 
the user is able to experience a sense of presence as if the 
user were actually present in the virtual space provided by 
the immersion image. 
5 The present inventors recognized several deficiencies 

with regard to the above -described conventional systems and 
methods. For example, since the display apparatus including 
the immersion screen, described above, displays an image 
that would be perceived by the vision of the user assuming 

10 that the user were in a virtual space, it is sometimes 

difficult for the user to grasp a relative position of the 
user in the virtual space provided by the immersion image. 

As another example, in an automotive racing game (i.e., 
a racing video game), a bird's-eye image of an automobile 

15 driven by a user is displayed as viewed from a certain 

viewpoint. In this case, the user is able to readily grasp 
the position of the automobile the user is driving, and thus 
drive the automobile in a more realistic manner. 

However, in a case where a bird's-eye image is 

20 displayed, compared with a case where an immersion image is 
displayed, a sense of immersion and a sense of presence are 
diminished . 

SUMMARY OF THE INVENTION 
25 The present invention has been made in view of the 
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above limitations with conventional apparatuses and methods. 
The present invention enables a user to readily grasp 
his/her own positional relationship and also allows an image 
with a sense of presence to be displayed. 
5 A display apparatus according to the present invention 

includes a first display mechanism for displaying a first 
image, which is disposed so as to surround the user; and a 
second display mechanism for displaying a second image, 
which is disposed inside the first display mechanism. 

10 An image processing apparatus according to the present 

invention includes an image selecting mechanism for 
selecting a viewpoint image of a predetermined object as 
viewed from a viewpoint of a user, from among images 
captured respectively from a plurality of positions, based 

15 on the viewpoint of the user; and a viewpoint -image 

converting mechanism for converting the viewpoint image into 
a light -emission image based on the viewpoint of the user 
and a shape of the display mechanism. 

An image processing method according to the present 

20 invention includes an image selecting step of selecting a 
viewpoint image of a predetermined object as viewed from a 
viewpoint of a user, from among images captured respectively 
from a plurality of positions, based on the viewpoint of the 
user; and a viewpoint - image converting step of converting 

25 the viewpoint image into a light -emission image based on the 
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viewpoint of the user and a shape of display mechanism. 

A program according to the present invention includes 
an image selecting step of selecting a viewpoint image of a 
predetermined object as viewed from a viewpoint of a user, 
5 from among images captured respectively from a plurality of 
positions, based on the viewpoint of the user; and a 
viewpoint -image converting step of converting the viewpoint 
image into a light -emission image based on the viewpoint of 
the user and a shape of display mechanism. 

10 An imaging apparatus according to the present invention 

includes an omnidirectional imaging mechanism for imaging an 
object's surroundings in all directions, and a plurality of 
object -imaging mechanisms for imaging the object from a 
plurality of directions. 

15 According to the display apparatus of the present 

invention, the first display mechanism that is disposed so 
as to surround (or at least partially surround, such as in a 
hemispherical context) a user displays a first image, and 
the second display mechanism disposed inside (e.g., within 

20 the hemispherical space) the first display mechanism 
displays a second image. 

According to the image processing apparatus, image 
processing method, and program of the present invention, a 
viewpoint image of a predetermined object as viewed from a 

25 viewpoint of a user is selected based on the viewpoint of 
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the user, and the viewpoint image is converted into a light - 
emission image based on the viewpoint of the user and a 
shape of display mechanism. 

According to the imaging apparatus of the present 
5 invention, surroundings in all directions around an object 
are imaged, and the object is imaged from a plurality of 
directions . 

Brief Description of the Drawings 

Fig. 1: Diagram showing an example configuration of an 
10 imaging/display system according to an embodiment of the 
present invention. 

Fig. 2: Block diagram showing an example configuration of 
an imaging system. 

Fig. 3: Perspective view showing an example configuration 
15 of a wide-angle imaging unit. 

Fig. 4: Perspective view showing an example configuration 
of a surrounding imaging unit. 

Fig. 5: Diagram illustrating a relationship of positions of 
an omnidirectional camera and surrounding cameras. 
20 Fig. 6: Block diagram showing an example configuration of a 
display system. 

Fig. 7: Perspective view showing an example configuration 
of a wide -angle -image display unit. 

Fig. 8: Perspective view showing an example configuration 
25 of a surrounding- image display unit. 
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Fig. 9: Diagram illustrating the relationship of positions 
of an outer dome screen and an inner dome screen. 
Fig. 10: Diagram illustrates a case where a plurality of 
users are viewing images. 

Fig. 11: Diagram showing a case where images of an object 
as viewed from respective viewpoints of a plurality of users 
are presented to the users. 

Fig. 12: Diagram showing a horizontal parallax system. 

Fig. 13: Diagram showing a vertical -and-horizontal parallax 

system. 

Fig. 14: Block diagram showing an example configuration of 
a wide r angle -image processing unit. 

Fig. 15: Diagram showing a wide -angle -imaging coordinate 
system. 

Fig. 16: Diagram showing an omnidirectional image. 

Fig. 17: Diagram showing an omnidirectional image and a 

longitude-latitude image . 

Fig. 18: Flowchart for explaining wide -angle -image 
processing . 

Fig. 19: Diagram for explaining an image that is displayed 
on an inner dome screen. 

Fig. 20: Diagram showing an image that is displayed on a 
display surface of an inner omni -directional projector and 
an image that is displayed on the inner dome screen. 
Fig. 21: Diagram showing a case where viewpoints of users A 
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and B are remote from each other. 

Fig. 22: Diagram showing a case where viewpoints of users A 
and B are close to each other. 

Fig. 23: Block diagram showing an example configuration of 
5 a surrounding- image processing unit. 

Fig. 24: Diagram for explaining processing that is executed 
by a viewpoint -direction calculating unit. 

Fig. 25: Diagram for explaining processing that is executed 
by a display-image generating unit. 
10 Fig. 26: Diagram for explaining processing that is executed 
by the display- image generating unit. 

Fig. 27: Diagram for explaining processing that is executed 
by the display- image generating unit. 

Fig. 28: Flowchart for explaining surrounding- image 
15 processing . 

Fig. 29: Flowchart for explaining the surrounding- image 
processing . 

Fig. 30: Block diagram showing an example configuration of 
a computer according to an embodiment of the present 
2 0 invention. 
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DETAILED DESCRIPTION OF THE INVENTION 
Fig. 1 shows an example configuration of an 
imaging/display system (a system refers to a logical 
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combination of a plurality of apparatuses, regardless of 
whether the apparatuses are in the same enclosure) according 
to an embodiment of the present invention. 

An imaging system 1 captures images of an object and 
5 images in all directions around (except, perhaps, beneath) 
the object. The imaging system 1 records the image data 
obtained on a recording medium 3, for example, an optical 
disk, a magnetic disk, a magnetic tape, or a semiconductor 
memory, or sends the image data via a transmission medium 4, 

10 for example, a telephone circuit, optical fiber, a satellite 
circuit, a microwave or other wireless link, a CATV (cable 
television) network, or the Internet. 

A display system 2 plays back image data from the 
recording medium 3, or receives and displays image data 

15 transmitted via the transmission medium 4. 

Fig. 2 shows an example configuration of the imaging 
system 1 shown in Fig. 1. A wide-angle imaging unit 11 
includes a wide-angle camera (video camera) for imaging the 
surroundings of the object over a wide angle, and it 

20 supplies wide -angle -image data (video data) captured by the 
wide-angle camera to a multiplexer 13. The wide-angle 
camera constituting the wide-angle imaging unit 11 is 
implemented, for example, by an omnidirectional camera for 
capturing images in all directions (or substantially all 

25 directions, such as all directions except directly beneath 
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the object, or even all directions with a field of view, or 
still all directions with a substantially hemispherical 
volume around the object) around the object, as will be 
described later. Thus, the wide-angle image data output 
5 from the wide-angle imaging unit 11 includes images such as 
the scenery of the surroundings of the object. 

A surrounding imaging unit 12 includes a large number 
of cameras (video cameras) disposed so as to surround the 
object in horizontal and vertical directions, and it 

10 supplies image data (video data) captured by the plurality 
of cameras to the multiplexer 13. In the surrounding 
imaging unit 12, the cameras disposed at a plurality of 
positions respectively output image data obtained by imaging 
the object from the positions of the cameras (camera 

15 positions) . The image data of the object, captured from the 
plurality of positions, will be collectively referred to as 
surrounding- image data when appropriate. 

The multiplexer 13 multiplexes the wide -angle -image 
data supplied from the wide-angle imaging unit 11 and the 

2 0 surrounding- image data supplied from the surrounding imaging 
unit 12, and supplies the resulting multiplexed image data 
to a recording/sending unit 14. 

More specifically, the multiplexer 13 multiplexes wide- 
angle- image data and surrounding- image data of the same 

25 frame in association with each other. Also, the multiplexer 
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13 multiplexes each frame of surrounding- image data, i.e., 
image data captured from the plurality of camera positions, 
in association with the camera positions. It is to be 
assumed that the camera positions of the respective cameras 
5 constituting the surrounding imaging unit 12 are known in 

advance. The camera positions are represented, for example, 
by coordinates representing positions on a three-dimensional 
coordinate system having an origin at the position of the 
obj ect . 

10 The recording/sending unit 14 records the multiplexed 

image data supplied from the multiplexer 13 on the recording 
medium 3 (Fig. 1) , or sends the multiplexed image data via 
the transmission medium 4 . 

Fig. 3 shows an example configuration of the wide-angle 

15 imaging unit 11 shown in Fig. 2. The wide-angle imaging 

unit 11 includes an omnidirectional camera 21. Ideally, the 
omnidirectional camera 21 receives light beams that are 
incident on the omnidirectional camera 21 through a 
hemispherical surface as shown in Fig. 3, and outputs image 

20 data corresponding to the light beams. 

The omnidirectional camera 21 is implemented, for 
example, by a camera with a fish-eye lens, or a camera with 
a hyperboloid mirror. In this embodiment, the 
omnidirectional camera 21 is implemented by a camera with a 

25 fish-eye lens. The resultant image of the object is 
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captured in substantially all directions with respect to an 
observation point, which in this case is the location of the 
omnidirectional camera . 

The omnidirectional camera 21 is disposed so as to 
5 allow imaging of the scenery, etc. in all directions around 
the object. The use of the term "all directions" is used 
herein in a general sense and is not intended to be narrowly 
interpreted; for instance the direction immediately beneath 
the camera or in the shadow of the object may not be 

10 included. For example, if the object is a soccer match that 
is being played in a stadium, the omnidirectional camera 21 
is disposed, for example, at a center of the stadium, so 
that the omnidirectional camera 21 captures images of the 
scenery of the stand where spectators are watching the 

15 soccer match, or even the sky, or inside of the stadium dome. 
Fig. 4 shows an example configuration of the 
surrounding imaging unit 12 shown in Fig. 2. The 
surrounding imaging unit 12 includes a large number of 
surrounding cameras 22 k (k = 1, 2, ...) that are disposed 

2 0 around the object so as to surround the object, so that the 
surrounding imaging unit 12 captures images of the object as 
viewed from a large number of viewpoints. Thus, for example, 
if the object is a soccer match that is being played in a 
stadium as described earlier, the large number of 

2 5 surrounding cameras 22 k constituting the surrounding imaging 
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unit 12 are respectively disposed so as to surround (i.e., 
at least within the gamut of candidate venues for watching 
the soccer field) a soccer field provided in the stadium, so 
that the surrounding cameras 2 2 k captures images of what is 
5 going on in the soccer match on the soccer field. 

The surrounding cameras 2 2 k may be disposed in a 
horizontal plane only in longitudinal directions (horizontal 
directions) of the object or in all longitudinal and 
latitudinal directions (horizontal and vertical directions) 

10 of the object. If the surrounding cameras 22 k are disposed 
only in the longitudinal directions of the object in a 
horizontal plane, only a horizontal parallax system, which 
will be described later, can be employed. If the 
surrounding cameras 22 k are disposed in all longitudinal and 

15 latitudinal directions of the object, either a horizontal 

parallax system or a vertical -and-horizontal parallax system, 
which will be described later, can be employed. 

Fig. 5 shows the relationship of positions where the 
omnidirectional camera 21 (Fig. 3) and the surrounding 

20 cameras 22 k (Fig. 4) are disposed. The surrounding cameras 
22 k are disposed, for example, on a spherical surface of a 
hemisphere surrounding the object, at regular longitudinal 
and latitudinal intervals. The surrounding cameras 22 k 
image the object with the positions of the surrounding 

25 cameras 22 k (camera positions) as viewpoints. 
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The omnidirectional camera 21 is disposed, for example, 
at a center of the hemisphere where the surrounding cameras 
22 k are disposed. The omnidirectional camera 21 images the 
surroundings of the object in all directions, i.e., in 
5 directions of points located on the curved surface of the 
hemisphere where the surrounding cameras 22 k are disposed 
from the center of the hemisphere. Moreover, the term "in 
all directions' 7 refers to directions that correspond to 
points on a surface from which there are candidate vistas of 

10 at least a portion of a space in which an object is located. 
This surface may also include the planar surface portion of 
the hemispherical shape, but need not (although may) include 
directions that are outside the field of view of the 
omnidirectional camera 21. 

15 In the embodiment shown in Fig. 5, the omnidirectional 

camera 21 is positioned so that an optical axis thereof 
extends in the direction of a north pole of the hemisphere 
where the surrounding cameras 22 k are disposed (in the 
direction of a point with latitude of 90 degrees) . The 

20 surrounding cameras 22 k are disposed so that optical axes 
thereof extend in the direction of the center of the 
hemisphere . 

It is to be understood that the omnidirectional camera 
21 and the surrounding cameras 22 k may be disposed at 
25 positions other than the positions shown in Fig. 5, or even 
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in spaces other than in a hemispherical space, such as a 
cube, or other traditional or non- traditionally defined 3 
dimensional interior space (e.g., it may be an arbitrarily 
shaped space) . 

5 In the imaging system 1 shown in Fig. 2, in the wide- 

angle imaging unit 11, the omnidirectional camera 21 (Fig. 
3) captures images in all directions around the object, and 
supplies obtained omnidirectional -image data (wide-angle- 
image data) to the multiplexer 13. At the same time, in the 

10 surrounding imaging unit 12, the surrounding cameras 22 k 
(Fig. 4) capture images of the object as viewed from the 
camera positions thereof, and supplies surrounding- image 
data obtained to the multiplexer 13 . 

The multiplexer 13 multiplexes the omnidirectional- 

15 image data supplied from the wide-angle imaging unit 11 and 
the surrounding- image data supplied from the surrounding 
imaging unit 12, and supplies the resulting multiplexed 
image data to the recording/sending unit 14. 

The recording/ sending unit 14 records the multiplexed 

2 0 image data supplied from the multiplexer 13 on the recording 
medium 3 (Fig. 1) , or sends the multiplexed image data to a 
destination (such as a computer at a remote location) via 
the transmission medium 4. 

Thus, in the imaging system 1, for example, if the 

25 object is a soccer match that is being played in a stadium, 
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the scenery of the stands where spectators are watching the 
soccer match, and a soccer ball kicked up high, etc. are 
imaged to obtain omnidirectional - image data, and images of 
the ground and players in the soccer match as viewed from a 
5 large number of viewpoints are captured to obtain 
surrounding - image data . 

Next, Fig. 6 shows an example configuration of the 
display system 2 shown in Fig. 1. A playback/receiving unit 
31 plays back multiplexed image data from the recording 

10 medium 3 (Fig. 1) , or receives multiplexed image data 

transmitted via the transmission medium 4 (Fig. 1) , and 
supplies the multiplexed image data to a demultiplexer 32. 

The demultiplexer 32 demultiplexes the multiplexed 
image data supplied from the playback/receiving unit 31 into 

15 frame -by- frame omnidirectional - image data and surrounding- 
image data, and supplies the omnidirectional -image data to a 
wide -angle -image processing unit 33 while supplying the 
surrounding -image data to a surrounding- image processing 
unit 35. 

20 The wide-angle- image processing unit 33 processes the 

omnidirectional -image data supplied from the demultiplexer 
32, and supplies the resulting image data to a wide-angle- 
image display unit 34. 

The wide-angle- image display unit 34, based on an image 

2 5 corresponding to the image data supplied from the wide- 
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angle-image processing unit 33, displays an immersion image 
that presents a user with a sense of immersion. 

The surrounding- image processing unit 3 5 processes the 
surrounding- image data supplied from the demultiplexer 32 in 
5 accordance with a user's viewpoint supplied from a viewpoint 
detecting unit 37, thereby generating image data of the 
object as viewed from the user's viewpoint, and supplies the 
image data to a surrounding- image display unit 36. 

The surrounding- image display unit 36, based on the 

10 image data supplied from the surrounding- image processing 

unit 35, displays a bird's-eye image of the object as viewed 
from a particular viewpoint such as the user's viewpoint. 

The viewpoint detecting unit 37 detects a viewpoint of 
a user, and supplies the user's viewpoint to the 

15 surrounding- image processing unit 35. The viewpoint 

detecting unit 37 is implemented, for example, by a magnetic 
sensor from Polhemus, Inc., which functions as a three- 
dimensional position sensor. In that case, the viewpoint 
detecting unit 37 implemented by the magnetic sensor is 

2 0 mounted, for example, on the head of the user. 

Alternatively, the viewpoint detecting unit 3 7 may be 
implemented by a mechanism that allow user-controlled input 
regarding (change of) a viewpoint, such as a joystick or a 
track ball. That is, the viewpoint detecting unit 37 need 

25 not allow detection of an actual viewpoint of a user, and 
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may be a mechanism that allows input of a virtual viewpoint 
of a user. 

Fig. 7 shows an example configuration of the wide- 
angle- image display unit 34 shown in Fig. 6. The wide- 
5 angle- image display unit 34 includes an outer 

omnidirectional projector 41 and an outer dome screen 42. 
The outer omnidirectional projector 41 is capable of 
emitting light beams over a wide angle, and is implemented, 
for example, by an ordinary projector and a fish-eye lens 

10 serving as an optical system that spreads light beams 
emitted by the projector over a wide angle. 

The outer dome screen 42 has a shape of, for example, a 
hemispherical dome, and is disposed so as to surround the 
user. Furthermore, the hemispherical dome serving as the 

15 outer dome screen 42 is large enough to allow a plurality of 
users to be accommodated therein and to move around to a 
certain extent, and the diameter thereof is, for example, on 
the order of 5 m. 

On an inner (or outer) surface of the outer dome screen 

20 42, for example, a coating material that causes light to be 
scattered is applied. Inside the outer dome screen 42, the 
outer omnidirectional projector 41 is disposed so that light 
beams will be emitted towards the inner surface of the outer 
dome screen 42, which in this hemispherical space 

25 constitutes "all directions' 7 . 
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More specifically, for example, the outer 
omnidirectional projector 41 is disposed so that an optical 
center thereof coincides with a center of the hemispherical 
dome serving as the outer dome screen 42 and so that an 
5 optical axis thereof is perpendicular to a horizontal plane. 
The outer omnidirectional projector 41, using the fish-eye 
lens thereof, emits light beams corresponding to the image 
data supplied from the wide-angle- image processing unit 3 3 
towards the entire inner surface of the outer dome screen 42 

10 having a shape of a hemispherical dome. 

The light beams emitted by the outer omnidirectional 
projector 41 are received and scattered at points on the 
inner surface of the outer dome screen 42 having a shape of 
a hemispherical dome. Thus, on the inner surface of the 

15 outer dome screen 42 disposed over the user, an immersion 

image corresponding to the light beams emitted by the outer 
omnidirectional projector 41, i.e., the omnidirectional 
image of the object captured by the imaging system 1, is 
displayed. 

20 The shape of the outer dome screen 42 is not limited to 

a hemispherical dome shape as long as the shape is such that 
the user is surrounded, and may be, for example, a cylinder 
shape, a cube shape, or another 3 -dimensional shape that 
does not have a commonly recognized name. 

2 5 Next, Fig. 8 shows an example configuration of the 
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surrounding- image display unit 36 shown in Fig. 6. The 
surrounding- image display unit 35 includes an inner 
omnidirectional projector 43 and an inner dome screen 44. 

The inner omnidirectional projector 43, similar to the 
5 outer omnidirectional projector 41 shown in Fig. 7, is 

capable of emitting light beams over a wide angle, and is 
implemented, for example, by an ordinary projector and a 
fish-eye lens serving as an optical system that spreads 
light beams emitted by the projector over a wide angle. 

10 The inner dome screen 44 has a predetermined shape of, 

for example, a hemispherical dome. On an inner (or outer) 
surface of the inner dome screen 44, for example, a coating 
material that causes light to be scattered is applied. 
Inside the inner dome screen 44, the inner omnidirectional 

15 projector 43 is disposed so that light beams will be emitted 
towards the inner surface of the inner dome screen 44 . 

More specifically, for example, the inner 
omnidirectional projector 43 is disposed so that an optical 
center thereof coincides with a center of the hemispherical 

2 0 dome serving as the inner dome screen 44 and so that an 

optical axis thereof is perpendicular to a horizontal plane. 
The inner omnidirectional projector 43, using the fish-eye 
lens thereof, emits light beams corresponding to the image 
data supplied from the wide -angle -image processing unit 33 

25 towards the entire inner surface of the inner dome screen 44 
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having a shape of a hemispherical dome. 

The light beams emitted by the inner omnidirectional 
projector 43 are received and scattered at points on the 
inner surface of the inner dome screen 44 having a shape of 
5 a hemispherical dome. The inner dome screen 44 itself is 

composed of a transparent material. The light beams emitted 
by the inner omnidirectional projector 43 are scattered on 
the inner surface of the inner dome screen 44, whereby a 
bird's eye image corresponding to the light beams emitted by 

10 the inner omnidirectional projector 43, i.e., an image of 
the object as viewed from a certain viewpoint, captured by 
the imaging system 1, is displayed on the outer surface of 
the inner dome screen 44 by what is called back projection. 
Although the shape of the inner dome screen 44 in this 

15 example is a hemispherical dome, which is similar to the 

shape of the outer dome screen 42 shown in Fig. 7, the shape 
of the inner dome screen 44 need not be similar to the shape 
of the outer dome screen 42. That is, the shape of the 
inner dome screen 44 may be, for example, a cylindrical 

2 0 shape, a polygonal -prism shape, or another shape that does 
not have a commonly recognizable name, independent of the 
shape of the outer dome screen 42 . 

Fig. 9 shows the relationship of positions where the 
outer dome screen 42 (Fig. 7) and the inner dome screen 44 

25 are disposed. As described earlier, the outer dome screen 
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42 is disposed so as to surround the user, and the inner 
dome screen 44 is disposed inside the outer dome screen 42. 
The use of the term "surround" in this context means that 
the outer dome screen 42 need not necessarily encapsulate 
5 the user, but merely give the user a sense of being 

contained within the space defined by the outer dome screen 
42. For example, the outer dome screen 42 preferable rests 
on the same surface that the user is standing on. However, 
the outer dome screen 42 may also be suspended, or supported 

10 in another way so that the outer dome screen 42 is elevated 
by some extent (e.g., an inch or two, up to a few feet). 

Thus, the user in a region inside the outer dome screen 
42 and outside the inner dome screen 44 is allowed to 
simultaneously view the immersion image displayed on the 

15 inner surface of the outer dome screen 42 and the bird's-eye 
image displayed on the outer surface of the inner dome 
screen 44. Accordingly, the user is allowed to experience a 
sense of immersion provided by the immersion image and to 
readily grasp his/her own position by the bird's-eye image. 

20 More specifically, for example, an immersion image of 

the scenery of the stand of the soccer stadium is displayed 
on the inner surface of the outer dome screen 42, and what 
is going on in the match on the soccer field as viewed from 
a viewpoint of the user is displayed on the outer surface of 

25 the inner dome screen 44. Accordingly, the user is allowed 
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to experience a sense of presence as if the user is actually 
watching the soccer match in the soccer stadium. 

As shown in Fig. 10, not only one, but a plurality of 
users can be accommodated inside the outer dome screen 42 . 
5 Thus, with the display system 2, a plurality of users is 

allowed to simultaneously view the immersion image displayed 
on the outer dome screen 42 and the bird's-eye image 
displayed on the inner dome screen 44. 

For example, if two users A and B are present inside 

10 the outer dome screen 42 and if the users A and B are at 
different positions as shown in Fig. 11, the inner dome 
screen 44 presents the user A with a bird's-eye image of the 
object as viewed from a viewpoint of the user A while 
presenting the user B with a bird's-eye image of the object 

15 as viewed from a viewpoint of the user A. 

That is, in the display system 2, the inner dome screen 
44 presents each user with a bird's-eye image of the object 
as viewed from a viewpoint of the user. 

Thus, the user is allowed to view an image (bird's-eye 

20 image) of the object as viewed from his/her viewpoint, and 
has the freedom for selecting a viewpoint. Viewpoint 
positions that can be selected by the user may be, for 
example, limited to positions in a horizontal plane at a 
certain height inside the outer dome screen 42, as shown in 

2 5 Fig. 12, or may be, for example, any positions inside the 
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outer dome screen 42 without such a limitation, as shown in 
Fig. 13. The system in which viewpoints are limited to 
positions on a certain horizontal plane, as shown in Fig. 12, 
can be called a horizontal parallax system since the user is 
5 allowed to move his/her viewpoint only in horizontal 

directions. The system in which viewpoints may be at any 
position, as shown in Fig. 13, can be called a vertical-and- 
horizontal parallax system since the user is allowed to move 
his/her viewpoint in any vertical and horizontal direction. 

10 Fig. 14 shows an example configuration of the wide- 

angle -image processing unit 3 3 shown in Fig. 6. The frame - 
by- frame omnidirectional -image data output from the 
demultiplexer 32 is supplied to a frame memory 51. The 
frame memory 51 stores, on a frame -by- frame basis, the 

15 omnidirectional -image data supplied from the demultiplexer 
32, and reads the omnidirectional image data on a frame-by- 
frame basis and supplies it to a wide -angle -image converting 
unit 52. The frame memory 51 includes a plurality of memory 
banks, and it allows simultaneous storing and reading of 

2 0 omnidirectional -image data by switching of the banks. 

The wide -angle -image converting unit 52, under the 
control of a controller 56, converts the omnidirectional - 
image data supplied from the frame memory 51 into latitude- 
longitude -image data representing a rectangular region on a 

25 plane defined by a latitudinal direction and a longitudinal 
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direction, and supplies it to an image-angle correcting unit 
53 . 

The image-angle correcting unit 53, under the control 
of the controller 56, corrects the latitude-longitude-image 
5 data supplied from the wide-angle-image converting unit 52, 
based on optical characteristics of the outer 
omnidirectional projector 41 of wide-angle- image display 
unit 34 (Fig. 7) , and supplies the latitude-longitude-image 
data having been corrected to a latitude- longitude- image 

10 converting unit 54. 

The latitude-longitude-image converting unit 54, under 
the control of the controller 56, converts the latitude- 
longitude- image data supplied from the image-angle 
correcting unit 53 into omnidirectional -image data, and 

15 supplies it to a frame memory 55. 

The frame memory 55 stores, on a frame -by- frame basis, 
the omnidirectional -image data supplied from the latitude- 
longitude -image converting unit 54, and reads the 
omnidirectional -image data on a frame -by- frame basis and 

20 supplies it to the outer omnidirectional projector 41 of the 
wide-angle- image display unit 34 (Fig. 7) . The frame memory 
55, similarly to the frame memory 51, includes a plurality 
of memory banks, and it allows simultaneous storing and 
reading of omnidirectional -image data by switching of the 

2 5 banks. 
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The controller 56, based on the status of storage of 
omnidirectional -image data in the frame memory 51, controls 
the wide-angle-image converting unit 52, the image-angle 
correcting unit 53, and the latitude-longitude- image 
5 converting unit 54. 

Next, processing that is executed by the wide-angle- 
image converting unit 52, the image-angle correcting unit 53, 
and the latitude- longitude- image converting unit 54 shown in 
Fig. 14 will be described with reference to Figs. 15 to 17. 

10 The wide-angle-image converting unit 52 receives 

omnidirectional -image data stored in the frame memory 51, 
i.e., omnidirectional -image data captured by the 
omnidirectional camera 21 constituting the wide-angle 
imaging unit 11 shown in Fig. 3. 

15 Now, letting a three-dimensional coordinate system 

shown in Fig. 15, in which an optical axis of the 
omnidirectional camera 21 constitutes a z axis and in which 
an imaging surface (photoreceptor surface) of the 
omnidirectional camera 21 constitutes an xy plane, be 

20 referred to as a wide-angle- imaging coordinate system, the 
relationship between light beams that are incident on the 
omnidirectional camera 21 and omnidirectional - image data 
captured by the omnidirectional camera 21 receiving the 
light beams will be considered in the wide-angle- imaging 

25 coordinate system. 
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The light beams L emitted toward the omnidirectional 
camera 21 are refracted by the fish-eye lens thereof and 
becomes incident on the imaging surface of the 
omnidirectional camera 21. That is, the fish-eye lens 
5 refracts the light beams L in a plane including the Z axis 
of the wide -angle -imaging coordinate system and the light 
beams L so that the light beams L will be incident on the xy 
plane. Thus, the omnidirectional camera 21 captures, as 
omnidirectional -image data, image data representing, for 

10 example, a circular region or a doughnut -shaped region shown 
as hatched in Fig. 16. 

Now, with regard to the wide-angle- imaging coordinate 
system, let an angle with respect to the x axis on the xy 
plane be referred to as a longitude 0, an angle with respect 

15 to the z axis be referred to as a latitude <)>, and the 

direction of a point with the longitude 9 and the latitude § 
be denoted as a direction (0, <|>) , as shown in Fig. 15. Then, 
a light beam L that is incident on the omnidirectional 
camera 21 from the direction (0, (|>) is refracted by the 

20 fish-eye lens of the omnidirectional camera 21 in accordance 
with the latitude (|>. 

Let the light beam L that is incident from the 
direction (0, be denoted as a light beam L(0, <J>) , and a 
point on the xy plane to which the light beam L(0, (()) is 

25 projected as Q(x, y) . Then, the longitude 0 and the 



- 27 - 



latitude can be expressed by the following equations using 
the x and y coordinates of the point Q. 

G = tan" 1 ~y 

(i) 

5 

0 = fi (Vx 2 +y 2 ) 

(2) 

10 In equation (2), the function f 1 () is determined 

according to optical characteristics of the fish-eye lens of 
the omnidirectional camera 21. Letting the inverse function 
of the function f x () be denoted as f 1 " 1 () / equation (2) can 
be rewritten as V (x 2 + y 2 ) = f 1 ' 1 (0) . A fish-eye lens for 

15 which the inverse function f -l^ 1 (9) , representing the optical 
characteristics of the fish-eye lens, can be expressed as 
(the focal length of the fish-eye lens) x 0 is called an 
equidistance projection method. 

From equations (1) and (2), a point Q(x, y) of the 

20 omnidirectional image captured by the omnidirectional camera 
21 can be expressed using a longitude 0 and a latitude (|) 
representing a direction of a light beam L. (0, <j)) that is 
incident on that point . 

More specifically, an angle that is formed by a line 
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connecting the point Q(x, y) and the origin 0 with respect 
to the x axis is equal to the longitude 9 of the light beam 
L. The distance between the point Q(x, y) and the origin O, 
V(x 2 -i- y 2 ) , depends on the latitude § of the light beam L. 
5 The longitude 0 and the latitude § are values independent of 
each other. 

Thus, for example, assuming a latitude-longitude 
coordinate system that is a two-dimensional coordinate 
system with the longitude 9 on the horizontal axis and the 
10 latitude <|) on the vertical axis, a pixel Q(x, y) 

constituting the omnidirectional -image data can be projected 
to a point Q 1 (9, (|>) on the latitude-longitude coordinate 
system. 

By projecting the omnidirectional - image data onto the 
15 latitude-longitude coordinate system, the omnidirectional - 
image data representing a doughnut - shaped region is 
converted into image data representing a rectangular region 
defined by the longitude 0 and the latitude <(>, i.e., 
latitude-longitude-image data, as shown in Fig. 17. 
20 The wide-angle-image converting unit 52 shown in Fig. 

14 converts omnidirectional - image data into latitude- 
longitude- image data in the manner described above. 

If the optical characteristics of the fish-eye lens of 
the omnidirectional camera 21 of the wide-angle imaging unit 
25 11 (Fig. 3) for capturing an omnidirectional image are the 



same as the optical characteristics of the fish-eye lens of 
the outer omnidirectional projector 41 of the wide-angle- 
image display unit 34 (Fig. 7) for emitting light beams 
corresponding to the omnidirectional image, when 
5 omnidirectional -image data captured by the omnidirectional 
camera 21 is supplied to the outer omnidirectional projector 
41 as it is, the outer omnidirectional projector 41 emits 
light beams that are different from light beams that are 
incident on the omnidirectional camera 21 only in that the 

10 directions are opposite. Thus, theoretically, the outer 

dome screen 42 (Fig. 7) , which receives the light beams from 
the outer omnidirectional projector 41, displays the same 
image of the scenery, etc. as captured by the 
omnidirectional camera 21. 

15 However, the optical characteristics of the fish-eye 

lenses of the omnidirectional camera 21 (Fig. 3) and the 
outer omnidirectional projector 41 (Fig. 7) are not 
necessarily the same and may differ from each other. 

The point Q(x, y) on the imaging surface that a light 

2 0 beam L(0, incident on the omnidirectional camera 21 

reaches is related to the longitude 9 and the latitude (() as 
expressed in equations (1) and (2) . 

The longitude 0 expressed by equation (1) is not 
affected by the optical characteristics of the fish-eye lens. 

25 On the other hand, the latitude (j) expressed in equation (2) 
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is calculated using the function f 1 () f and since the inverse 
function f 1 " 1 () of the function f 1 () represents the optical 
characteristics of the fish-eye lens as described earlier, 
the latitude § is affected by the optical characteristics of 
5 the fish-eye lens. 

Thus, when the optical characteristics of the fish-eye 
lenses of the omnidirectional camera 21 (Fig. 3) and the 
outer omnidirectional projector 41 (Fig. 7) differ from each 
other, in order that the image displayed on the outer dome 

10 screen 42 (Fig. 7) be the same as the image of the scenery, 
etc. as captured by the omnidirectional camera 21, the 
omnidirectional -image data captured by the omnidirectional 
camera 21 must be corrected with respect to the direction of 
the latitude § . 

15 Since the direction of the latitude <[> of the 

omnidirectional - image data is along the direction of the 
vertical axis of the latitude-longitude-image data shown in 
Fig. 17, the image-angle correcting unit 53 shown in Fig. 14 
corrects the latitude § on the vertical axis of the 

20 latitude-longitude-image data supplied from the wide-angle- 
image converting unit 52 into a latitude § 1 , for example, 
according to the following equation. 



<t> 9 = Oi 0 

25 
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In equation (3) , a is a value that is determined by the 
optical characteristics of the fish-eye lenses of the 
omnidirectional camera 21 (Fig. 3) and the outer 
omnidirectional projector 41 (Fig. 7) . 
5 Thus, letting a pixel constituting the latitude- 

longitude- image data supplied from the wide-angle- image 
converting unit 52 be denoted as Q ' (0, <f>) , the image-angle 
correcting unit 53 corrects the pixel Q ! (8, ((>) into a pixel 
Q ' (6, <|>' ) . 

10 The latitude which represents the direction of the 

vertical axis of the latitude-longitude-image data as shown 
in Fig. 17, corresponds to an image angle in the vertical 
direction of the latitude-longitude-image data. Thus, the 
correction of the latitude <|> corresponds to increasing or 

15 decreasing the image angle in the vertical direction of the 
latitude -longitude -image data . 

If the latitude-longitude-image data representing a 
rectangular region is supplied as it is to the outer 
omnidirectional projector 41 (Fig. 7), the outer dome screen 

20 42 does not display the omnidirectional image captured by 

the omnidirectional camera 21 (Fig. 3) . Thus, the latitude- 
longitude- image converting unit 54 shown in Fig. 14 converts 
the latitude-longitude-image data with the latitude <j> 
thereof having been corrected in the image-angle correcting 

25 unit, back into omnidirectional-image data representing a 
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doughnut -shaped region as shown in Fig. 16 (or a circular 
region) . In the omnidirectional -image data yielded by the 
latitude-longitude-image converting unit 54, the scale along 
the direction of the latitude § has been corrected based on 
5 the optical characteristics of the fish-eye lenses of the 
omnidirectional camera 21 (Fig. 3) and the outer 
omnidirectional projector 41 (Fig. 7) . Thus, when the 
omnidirectional -image data is supplied to the outer 
omnidirectional projector 41 (Fig. 7) , the outer dome screen 

10 42 displays the omnidirectional image captured by the 
omnidirectional camera 21 (Fig. 3) . 

Next, processing that is executed in the wide-angle- 
image processing unit 33 shown in Fig. 14 (wide -angle -image 
processing) will be described with reference to a flowchart 

15 shown in Fig. 18. 

The controller 56 monitors the frame memory 51, and it 
starts wide -angle -image processing when the frame memory 51 
starts storing omnidirectional -image data. 

More specifically, in the wide -angle -image processing, 

20 first in step SI, the controller 56 sets an initial value, 
for example, 1, in a variable i for counting the number of 
frames, and the processing then proceeds to step 32. In 
step S2, the frame memory 51 reads omnidirectional - image 
data of an i-th frame stored therein, and supplies it to the 

25 wide-angle- image converting unit 52. The processing then 
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proceeds to step S3 . 

In step S3, the wide-angle-image converting unit 52 
calculates, according to equation (1) , a longitude 0 for 
each pixel Q(x, y) constituting the omnidirectional -image 
5 data supplied from the frame memory 51, and the processing 
then proceeds to step S4 . In step S4 , the wide-angle- image 
converting unit 52 calculates, according to equation (2) , a 
latitude <|> for each pixel Q(x, y) constituting the 
omnidirectional -image data supplied from the frame memory 51, 

10 and the processing then proceeds to step S5. 

In step S5, the wide-angle- image converting unit 52 
maps each pixel Q(x, y) constituting the omnidirectional - 
image data supplied from the frame memory 51 to a point (0, 
(()) on the latitude -longitude coordinate system, represented 

15 by the longitude 0 and the latitude § that have been 

calculated for the pixel Q(x, y) , thereby converting the 
omnidirectional -image data representing a doughnut -shaped 
region into latitude-longitude-image data representing a 
rectangular region, as shown in Fig. 17. 

20 The wide-angle- image converting unit 52, having 

obtained the latitude-longitude-image data in step S5 as 
described above, supplies the latitude-longitude- image data 
to the image-angle correcting unit 53, and the processing 
then proceeds to step S6. 

25 In step S6, the image-angle correcting unit 53 corrects 
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the latitude <|> of the latitude-longitude-image data supplied 
from the wide-angle- image converting unit 52 into a latitude 
c)) 1 according to equation (3) , and supplies latitude- 
longitude -image data represented by the corrected latitude 
c|>' and the longitude 0 to the latitude-longitude-image 
converting unit 54. The processing then proceeds to step S7 . 

In step S7, the latitude -longitude -image converting 
unit 54 calculates, according to equation (4) (see below) , 
an x coordinate on the xy plane of the wide-angle-imaging 
coordinate system, for each pixel Q'(G, (J)') constituting the 
longitude-latitude-image data supplied from the image-angle 
correcting unit 53, and the processing then proceeds to step 
S8. 

In step S8, the latitude-longitude- image converting 
unit 54 calculates, according to equation (5) , ay 
coordinate on the xy plane of the wide-angle- imaging 
coordinate system, for each pixel Q"(9, <j> 1 ) constituting the 
longitude-latitude- image data supplied from the image-angle 
correcting unit 53, and the processing then proceeds to step 
S9 . 



X = f 2 (0')cOS0 



y = f 2 (0')sin0 



(4) 
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(5) 

The function f 2 ((|) f ) in equations (4) and (5) represents 
the optical characteristics of the outer omnidirectional 
projector 41 (Fig. 7) (a function that is determined by a 
5 projection method of the fish-eye lens of the outer 

omnidirectional projector 41) . Solving equations (1) and 
(2) yields equations similar to equations (4) and (5) . 

In step S9, the latitude -longitude -image converting 
unit 54 maps each pixel Q'(6, (() ■ ) constituting the 

10 longitude -latitude -image data supplied from the image -angle 
correcting unit 53 to a point (x, y) on the xy plane of the 
wide-angle-imaging coordinate system, represented by the x 
and y coordinates obtained for the pixel Q' (9, (()'), thereby 
converting the latitude -longitude -image data representing a 

15 rectangular region into omnidirectional - image data 

representing a doughnut - shaped region. The omnidirectional - 
image data is supplied to and stored in the frame memory 55, 
as light-emission omnidirectional - image data for causing the 
outer omnidirectional projector 41 (Fig. 7) to emit light. 

20 The light-emission omnidirectional - image data stored in 

the frame memory 55 as described above is sequentially read 
and supplied to the wide-angle-image display unit 34 (Fig. 
6). In the wide-angle- image display unit 34, the outer 
omnidirectional projector 41 (Fig. 7) emits light beams 

25 corresponding to the light-emission omnidirectional-image 
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data, and the outer dome screen 42 (Fig. 7) receives the 
light beams, whereby the outer dome screen 42 displays, as 
an immersion image, an image of the scenery in all 
directions around the object, captured by the 
5 omnidirectional camera 21 (Fig. 3) . 

Then, the processing proceeds to step S10, in which the 
controller 51 refers to the frame memory 51 to determine 
whether the i-th frame is the last frame. If it is 
determined in step S10 that the i-th frame is not the last 

10 frame, that is, if omnidirectional -image data of a frame 

subsequent to the i-th frame is stored in the frame memory 
51, the processing proceeds to step Sll, in which the 
controller 56 increments the variable i by 1 . The 
processing then returns to step S2 , and similar the same 

15 processing is repeated. 

On the other hand, if it is determined in step S10 that 
the i-th frame is the last frame, that is, if 

omnidirectional -image data of a frame subsequent to the i-th 
frame is not stored in the frame memory 51, the wide-angle- 

2 0 image process is stopped. 

If the omnidirectional camera 21 (Fig. 3) and the outer 
omnidirectional projector 41 (Fig. 7) have the same optical 
characteristics, omnidirectional -image data output by the 
demultiplexer 32 (Fig. 6) may be directly supplied to the 

25 wide -angle -image display unit 34 as light-emission 
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omnidirectional -image data, bypassing the wide-angle- image 
processing unit 3 3 shown in Fig. 14. 

Next, an overview of the surrounding- image processing 
unit 35 shown in Fig. 6 will be described. In the 
5 surrounding- image display unit 36 (Fig. 6), the inner dome 
screen 44 (Fig. 8) presents the user with an image of the 
object as viewed from a viewpoint of the user, as described 
earlier. If a plurality of users exists, the inner dome 
screen 44 (Fig. 8) presents the plurality of users with 

10 images of the object as viewed from the respective 
viewpoints of the users. 

Thus, if the inner dome screen 44 is transparent and if 
the object imaged by the surrounding imaging unit 12 (Fig. 
2) exists inside the inner dome screen 44, the inner dome 

15 screen 44 displays the same image of the object as perceived 
by the vision of each user, as shown in Fig. 19. 

More specifically, in the embodiment shown in Fig. 19, 
two users A and B are present, and the inner dome screen 44 
displays an image of the object as viewed from a viewpoint 

2 0 of the user A at a point where a straight line connecting 

the center of the inner dome screen 44 and the viewpoint of 
the user A intersects the inner dome screen 44. Furthermore, 
the inner dome screen 44 displays an image of the object as 
viewed from a viewpoint of the user B at a point where a 

25 straight line connecting the center of the inner dome screen 




44 and the viewpoint of the user B intersects the inner dome 
screen 44. 

Thus, in the surrounding display unit 3 6 shown in Fig. 
8, for example, as shown in Fig. 20, the inner 
5 omnidirectional projector 43 displays an image of the object 
as viewed from a viewpoint of each user (hereinafter 
referred to as a viewpoint image when appropriate) on a 
display surface of, for example, a liquid crystal panel that 
is not shown, and emits light beams corresponding to the 

10 viewpoint image through the fish-eye lens. Accordingly, the 
inner dome screen 44 presents each user with an image of the 
object as viewed from a viewpoint of the user. 

The surrounding- image processing unit 3 5 shown in Fig. 
6, using the surrounding- image data supplied from the 

15 demultiplexer 32, generates an image of the object as viewed 
from each viewpoint supplied from the viewpoint detecting 
unit 3 7 (viewpoint image) , and maps the viewpoint image 
associated with each viewpoint to a position in accordance 
with the viewpoint, thereby generating a light-emission 

2 0 image corresponding to light beams to be emitted by the 

inner omnidirectional projector 43 (Fig. 8), i.e., a light- 
emission image to be displayed on the display surface of the 
inner omnidirectional proj ector 4 3 (hereinafter also 
referred to as a display image) . 

25 When the users A and B are distant and viewpoints of 
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the users A and B are therefore distant from each other, for 
example, as shown in Fig. 21, even if the inner dome screen 
44 presents the users A and B with images of the object as 
viewed from their respective viewpoints (viewpoint images) , 
5 the user A is allowed to view only the viewpoint image 

associated with the viewpoint of the user A while the user B 
is allowed to view only the viewpoint image associated with 
the viewpoint of the user B. 

That is, the viewpoint image associated with the 

10 viewpoint of the user B is not seen by the user A, or is 
seen only to an extent such that viewing of the viewpoint 
image associated with the viewpoint of the user A is not 
interfered. Similarly, the viewpoint image associated with 
the user A is not seen by the user B, or is seen only to an 

15 extent such that viewing of the viewpoint image associated 
with the viewpoint of the user B is not interfered. 

On the other hand, when the users A and B are close to 
each other, for example, as shown in Fig. 22, their 
viewpoints are also close to each other, so that viewpoint 

2 0 images displayed on the inner dome screen 44 respectively 

for the users A and B could overlap each other. Even if the 
viewpoint images displayed on the inner dome screen 44 
respectively for the users A and B do not overlap each other, 
the viewpoint images for the users A and B are displayed in 

25 proximity to each other on the inner dome screen 44. Thus, 
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the viewpoint image associated with the viewpoint of the 
user B interferes with viewing by the user A of the 
viewpoint image associated with the viewpoint of the user A, 
and the viewpoint image associated with the viewpoint of the 
5 user A interferes with viewing by the user B of the 

viewpoint image associated with the viewpoint of the user B. 

Thus, when the distance between the viewpoints of the 
users A and B is short, the surrounding- image processing 
unit 3 5 sets a common viewpoint for the users A and B at a 

10 position close to both of the viewpoints of the users A and 
B, and generates an image of the object as viewed from that 
viewpoint (viewpoint image) . In that case, as shown in Fig. 
22, the inner dome screen 44 presents both of the users A 
and B with an image of the object as viewed from the common 

15 viewpoint, so that image viewing by the users A and B is not 
interfered with. 

Fig. 2 3 shows an example configuration of the 
surrounding- image processing unit 35 shown in Fig. 6, which 
executes the processing described above (surrounding -image 

20 processing) . A viewpoint buffer 61 receives a viewpoint of 

a user, output from the viewpoint detecting unit 37 (Fig. 6). 
The viewpoint buffer 61 stores the viewpoint supplied from 
the viewpoint detecting unit 37. If a plurality of users is 
present, the viewpoint detecting unit 3 7 shown in Fig. 6 

25 detects viewpoints of the plurality of users, and supplies 
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the viewpoints to the viewpoint buffer 61. In that case, 
the viewpoint buffer 61 stores all the viewpoints of the 
plurality of users, supplied from the viewpoint detecting 
unit 37. 

5 A viewpoint processing unit 62 reads the viewpoint of 

the user, stored in the viewpoint buffer 61, executes 
predetermined processing on the viewpoint as required, and 
supplies the result to a viewpoint storing unit 63. The 
viewpoint storing unit 63 stores the viewpoint supplied from 
10 the viewpoint processing unit 62. A viewpoint -direction 

calculating unit 64 calculates a direction of the viewpoint 
stored in the viewpoint storing unit 63 (viewpoint 
direction) , and supplies it to a viewpoint -image converting 
unit 67. 

15 A frame memory 65 stores, on a frame -by- frame basis, 

surrounding -image data supplied from the demultiplexer 32 
(Fig. 6) , and reads the surrounding- image data and supplies 
it to an image selecting unit 66. The frame memory 65 is 
constructed similarly to the frame memory 51 shown in Fig. 

20 14, so that the frame memory 65 allows simultaneous storing 
and reading of surrounding- image data. 

The image selecting unit 66 selects image data needed 
to generate a viewpoint image associated with the viewpoint 
stored in the viewpoint storing unit 63 from the 

25 surrounding- image data supplied from the frame memory 65. 
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More specifically, the surrounding -image data is a set of 
image data captured by the plurality of surrounding cameras 
22 x , 22 2 , 22 3 , ... disposed at a plurality of camera positions, 
as shown in Fig. 4, and letting the image data constituting 
5 the surrounding- image data, captured by a surrounding camera 
22 k/ be referred to as camera-image data, the image 
selecting unit 66 selects camera-image data from the 
surrounding -image data based on the viewpoint stored in the 
viewpoint storing unit 63. 

10 Then, the image selecting unit 66 supplies the camera- 

image data selected from the surrounding- image data based on 
the viewpoint stored in the viewpoint storing unit 63, as it 
is or after processing it, to the viewpoint- image converting 
unit 67 as viewpoint -image data associated with the 

15 viewpoint. 

The viewpoint -image converting unit 67, based on the 
viewpoint direction supplied from the viewpoint -direction 
calculating unit 64 and based on the shape of the inner dome 
screen 44 (Fig. 8), generates display-image data for 

2 0 displaying an image on the display surface of the inner 

omnidirectional projector 43 (Fig. 8) using the viewpoint 
image data supplied from the image selecting unit 66, and 
supplies it to a frame memory 68. 

The image selecting unit 66 and the viewpoint - image 

25 converting unit 67 described above constitute a display- 
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image generating unit 70 for generating display- image data 
from surrounding- image data based on the viewpoint of the 
user and the shape of the inner dome screen 44 (Fig. 8) . 
The frame memory 68 stores the display- image data 
5 supplied from the viewpoint - image converting unit 67, and 

supplies the display- image data to the inner omnidirectional 
projector 43 (Fig. 8) . The frame memory 68 is constructed 
similarly to the frame memory 51 shown in Fig. 14, so that 
the frame memory 68 allows simultaneous storing and reading 

10 of display- image data. 

A controller 69 controls the viewpoint processing unit 
62, the viewpoint-direction calculating unit 64, the image 
selecting unit 66, and the viewpoint -image converting unit 
67 with reference to the status of storage of surrounding- 

15 image data in the frame memory 65. 

Next, processing that is executed by the viewpoint- 
direction calculating unit 64 shown in Fig. 23 will be 
described with reference to Fig. 24. The viewpoint- 
direction calculating unit 64 calculates a longitude and a 

2 0 latitude that represent a direction of the viewpoint stored 
in the viewpoint storing unit 63 (viewpoint direction) . 

Now, letting a three-dimensional coordinate system in 
which the optical axis of the inner omnidirectional 
projector 43 constitutes a z axis and in which the display 

2 5 surface of the inner omnidirectional projector 43 
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constitutes an xy plane be referred to as a surrounding- 
image-display coordinate system, the viewpoint -direction 
calculating unit 64 calculates, as a viewpoint direction, 
the direction of a viewpoint V(x, y, z) at a certain 
5 position as viewed from the origin 0 in the surrounding- 
image-display coordinate system. 

Now, with regard to the surrounding- image -display 
coordinate system, letting an angle with respect to the x 
axis on the xy plane be referred to as a longitude 0, an 

10 angle with respect to the z axis be referred to as a 

latitude <|), and the direction of a point with a longitude 0 
and a latitude <|> be denoted as a direction (0, , the 
longitude 0 and the latitude <j) representing the viewpoint 
direction (0, <|>) of the viewpoint V(x, y, z) can be 

15 calculated by the following equations, respectively. 



6 = tan 



x 



(6) 



20 




(7) 
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The viewpoint -direction calculating unit 64 calculates 
the viewpoint direction (0, <|>) according to equations (6) 
and (7) , and supplies it to the viewpoint- image converting 
unit 67. 

5 Next, processing that is executed by the display- image 

generating unit 70 shown in Fig. 23 will be described with 
reference to Figs. 25 to 27. 

In order to present a user with a display on the inner 
dome screen 44 (Fig. 8) of a viewpoint image of an object as 

10 viewed from a viewpoint of the user, simply, for example, as 
shown in Fig. 25, camera -image data captured from a 
viewpoint direction (0, ()>) of the user's viewpoint must be 
displayed as a viewpoint image in a predetermined range on 
the inner dome screen 44, centered at an intersection U of 

15 the viewpoint direction (0, (|>) and the inner dome screen 44. 
Now, letting the viewpoint image, i.e., the camera- 
image data captured from the viewpoint direction (0, (|)) , be 
denoted as a viewpoint image (0, <j)) , and the predetermined 
range on the inner dome screen 44, centered at the 

20 intersection U of the viewpoint direction (0, (()) of the 
viewpoint of the user and the inner dome screen 44 as a 
display area (0, (|>, in order to display the viewpoint image 
(0, <|>) in the display area (0, ())) , the viewpoint image (0, <J>) 
must be written to a position on a display image to be 

25 displayed on the display surface of the inner 



omnidirectional projector 43 (Fig. 8) . 

Thus, in the display-image generating unit 70 shown in 
Fig. 23, the image selecting unit 66 selects the viewpoint 
image (9, <|>) , i.e., the camera-image data captured from the 
viewpoint direction (0, <()) of the viewpoint stored in the 
viewpoint storing unit 63, and the viewpoint -image 
converting unit 67 maps the viewpoint image (0, to a 
display image so that the viewpoint image (0, <|)) can be 
displayed in the display area (0, <j>) . 

Now, let a two-dimensional coordinate system in which 
the display surface of the inner omnidirectional projector 
43 constitutes an xy plane be referred to as a display- 
surface coordinate system, and an x coordinate and a y 
coordinate thereof as an x a coordinate and a y a coordinate, 
as shown in Fig. 26. Furthermore, let a two-dimensional 
coordinate system in which the viewpoint image (0, (|>) 
constitutes an xy plane be referred to as a viewpoint -image 
coordinate system. As shown in Fig. 26, the origin O a of 
the display-surface coordinate system is taken at an 
intersection of the optical axis of the inner 
omnidirectional projector 43 and the display surface, and 
the origin O of the viewpoint -image coordinate system is 
taken at a center of the rectangular viewpoint image (0, 
(intersection of diagonals of the rectangular viewpoint 
image (0, (J>) ) . 
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In order to display the viewpoint image (0, <|>) in the 
display area (0, (J)) , first, the viewpoint image (0, (J>) must 
be mapped to a position that is distant from the origin 0 a 
of the display-surface coordinate system by a distance g((|>) 
5 corresponding to the latitude as shown in Fig. 26. 

Furthermore, in order to display the viewpoint image (0, (|)) 
in the display area (0, (()) , second, the viewpoint image (0, 
<[)) must be rotated by an angle of (7t/2 + 0) counterclockwise,^ 
as shown in Fig. 27. 
10 Thus, the viewpoint -image converting unit 67 first 

calculates, according to equation (8) , coordinates (x ao , y ao ) 
in the display-surface coordinate system of the origin O of 
the viewpoint -image coordinate system of the viewpoint image 



(0, <|0 . 



15 




(8) 



20 



In equation (8), the function g(<|>) represents the 



optical characteristics of the inner omnidirectional 



projector 43 (Fig. 8) (a function that is determined by a 



projection method of the fish-eye lens of the inner 
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omnidirectional projector 43) . 

Furthermore, letting coordinates in the viewpoint -image 
coordinate system of a pixel Q constituting the viewpoint 
image (0, (()) be denoted as Q(x Q/ y Q ) , and coordinates thereof 
5 in the display-surface coordinate system as Q(x aQ/ y aQ ) , the 
viewpoint -image converting unit 67 calculates, according to 
equation (9), coordinates Q(x aQ , y aQ ) in the display-surface 
coordinate system of the pixel Q constituting the viewpoint 
image (0, (()) . 




(9) 



15 Then, the viewpoint -image converting unit 67 writes the 

pixel value of each pixel Q(x Q , y Q ) constituting the 
viewpoint image (0, <|)) in the frame memory 68, at a position 
corresponding to coordinates (x aQ/ y a o) expressed by equation 
(9) , thereby converting the viewpoint image (0, (|>) into a 

2 0 display image . 

Next, processing that is executed by the surrounding- 
image processing unit 3 5 shown in Fig. 2 3 (surrounding -image 
processing) will be described with reference to flowcharts 
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shown in Figs. 2 8 and 29. 

The controller 69 monitors the frame memory 65, and it 
starts surrounding- image processing when the frame memory 65 
starts storing surrounding- image data. 
5 More specifically, in the surrounding- image processing, 

first in step S21, the controller 69 sets an initial value, 
for example, 1, in a variable i for counting the number of 
frames, and the processing then proceeds to step S22 . 

The process of setting 1 in the variable i in step S21, 

10 and a process of incrementing the variable i in step S53, 

which will be described later, are executed respectively in 
synchronization with the process of setting 1 in the 
variable i in step SI and the process of incrementing the 
variable i in step Sll, described earlier and shown in Fig. 

15 18. That is, the wide-angle- image processing in the wide- 
angle- image processing unit 3 3 and the surrounding- image 
processing in the surrounding- image processing unit 3 5 are 
executed in synchronization on the same frame. Thus, the 
wide -angle -image display unit 34 and the surrounding- image 

20 display unit 3 6 display an immersion image and a bird's-eye 
image of the same frame in synchronization. 

In step S22, the controller 69 sets an initial value, 
for example, 1, in a variable n for counting the number of 
users, and the processing then proceeds to step S23 . In 

25 step S23, the viewpoint processing unit 62 reads the 
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viewpoint of a user #n from the viewpoint buffer 61, and 
registers it in the viewpoint storing unit 63 . 

More specifically, the viewpoint detecting unit 37 (Fig. 
6) detects viewpoints of all the users present inside the 
5 outer dome screen 42 (Fig. 7), for example, at a frame cycle, 
and updates contents stored in the viewpoint buffer 61 with 
the latest viewpoints detected. In step S23, the viewpoint 
processing unit 62 reads the viewpoint of the user #n, 
stored in the viewpoint buffer 61 at the timing of an i-th 

10 frame, and registers it in the viewpoint storing unit 63. 

The viewpoint of the user #n may be stored in the viewpoint 
storing unit 63 together with, for example, an ID 
(identification) representing the user #n. 

After the viewpoint processing unit 62 has registered 

15 the viewpoint of the user #n in the viewpoint storing unit 
63, the processing proceeds to step S24, in which the 
controller 69. determines whether the variable n equals the 
total number N of users present inside the outer dome screen 
42 (Fig. 7) . If it is determined in step S23 that the 

2 0 variable n does not equal N, the processing proceeds to step 
S25, in which the controller 69 increments the variable n by 
1. The processing then returns to step S23, and the same 
processing is repeated. 

On the other hand, if it is determined in step S24 that 

25 the variable n equals N, that is, when viewpoints of all the 
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users present inside the outer dome screen 42 (Fig. 7) have 
been stored in the viewpoint storing unit 63, the processing 
proceeds to step S26. In step S26, the controller 6 9 sets, 
as an initial value, the total number N of users present 
5 inside the outer dome screen 42 (Fig. 7) in a variable M 

representing the total number of viewpoints to be processed 
in the viewpoint selecting unit 66. The processing then 
proceeds to step S27. 

In step S27, the controller 69 sets an initial value, 

10 for example, 1, in a variable m for counting the number of 
viewpoints, and sets an initial value in a flag indicating 
the presence or absence of short-range viewpoints, for 
example, 0 indicating the absence of short-range viewpoints, 
which will be described later. The processing then proceeds 

15 to step S28. 

In step S28, the viewpoint processing unit 62, with 
regard to an m-th viewpoint among the M viewpoints stored in 
the viewpoint storing unit 63, calculates distances from the 
other (M - 1) respective viewpoints (viewpoint distances) , 

20 and detects viewpoints with viewpoint distances not greater 
than (or distances smaller than) a predetermined threshold 
value TH1 as short-range viewpoints for the m-th viewpoint. 
The processing then proceeds to step S2 9, in which the 
viewpoint processing unit 62 determines whether a short - 

25 range viewpoint exists for the m-th viewpoint. 
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If it is determined in step S2 9 that a short-range 
viewpoint does not exist for the m-th viewpoint, that is, if 
a viewpoint with a viewpoint distance from the m-th 
viewpoint not greater than the threshold value TH1 is not 
5 stored (registered) in the viewpoint storing unit 63, steps 
S3 0 to S34 are skipped and the processing proceeds to step 
S35 . 

On the other hand, if it is determined in step S2 9 that 
a short-range viewpoint exists for the m-th viewpoint, that 

10 is, if a viewpoint with a viewpoint distance from the m-th 
viewpoint not greater than the threshold value TH1 (short- 
range viewpoint) is stored (registered) in the viewpoint 
storing unit 63, the processing proceeds to step S30. In 
step S30, the controller 69 sets, for example, 1 in the flag, 

15 indicating the presence of a short-range viewpoints, and the 
processing then proceeds to step S31. 

The process in step S30 can be skipped if 1 is already 
set in the flag. 

In step S31, the viewpoint processing unit 62 detects 

20 the number of viewpoints determined as short-range 

viewpoints for the m-th viewpoint among the viewpoints 
stored in the viewpoint storing unit 63, and further detects 
a barycenter (a center of mass between two associated 
objects) of the short-range viewpoints and the m-th 

25 viewpoint. 
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Then, the processing proceeds to step S32, in which the 
viewpoint processing unit 62 registers (writes) the 
barycenter calculated in step S31 as the m-th viewpoint in 
the viewpoint storing unit 63 by overwriting, and the 
5 processing then proceeds to step S33. In step S33, the 
viewpoint processing unit 62 deletes the viewpoints 
determined as short-range viewpoints for the m-th viewpoint 
among the viewpoints stored in the viewpoint storing unit 63, 
and the processing then proceeds to step S34 . 

10 In step S34, the controller 69 subtracts the number of 

short-range viewpoints detected in step S31 from the 
variable M representing the total number M of viewpoints to 
be processed in the viewpoint selecting unit 66, and newly 
sets the resulting value of subtraction to the variable M, 

15 and the processing then proceeds to step S35. That is, 

since the short-range viewpoints among the viewpoints stored 
in the viewpoint storing unit 63 are deleted in step S33, 
the value of the variable M is decreased in step S34 by the 
number of short-range viewpoints deleted. 

20 Thus, by the processing from step S28 to step S34, 

enclosed by a dotted line in Fig. 28, one or more viewpoints 
that are close in distance to the m-th viewpoint are 
detected, and the one or more viewpoints and the m-th 
viewpoint are integrated into a position at the barycenter 

25 of these viewpoints. 
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In step S35, the controller 69 determines whether the 
variable m equals the variable M, i.e., the number of 
viewpoints M stored in the viewpoint storing unit 63. If it 
is determined in step S3 5 that the variable m does not equal 
the variable M, the processing proceeds to step S36, in 
which the controller 69 increments the variable m by 1 . The 
processing then returns to step S2 8, and the same processing 
is repeated. 

On the other hand, if it is determined in step S35 that 
the variable m equals the variable M, the processing 
proceeds to step S37, in which the controller 69 determines 
whether the flag equals 1. If it is determined in step S37 
that the flag equals 1, that is, if a viewpoint having been 
integrated in the previous loop processing from step S2 8 to 
step S36 exists among the viewpoints stored in the viewpoint 
storing unit 63 and if viewpoints could be further 
integrated, the processing returns to step S2 7, and the same 
processing is repeated. 

On the other hand, if it is determined in step S37 that 
the flag does not equal 1, that is, if the flag remains 0 as 
set in step S27 and if a viewpoint with a distance from 
other viewpoints not greater than the threshold value TH1 
does not exist among the viewpoints stored in the viewpoint 
storing unit 63, the processing proceeds to step S41 shown 
in Fig. 29. 
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In step S41, the controller 69 sets an initial value, 
for example, 1, to the variable m for counting the number of 
viewpoints, and the processing proceeds to step S42 . In 
step S42, the image selecting unit 66 calculates, as camera 
5 distances, distances between the m-th viewpoint stored in 
the viewpoint storing unit 66 and the camera viewpoints of 
the surrounding cameras 22 k constituting the surrounding 
imaging unit 12 . 

The camera viewpoints of the surrounding cameras 22 k 

10 refer to camera positions that are multiplexed with 
surrounding- image data, normalized (divided) by a 
predetermined normalization factor p. The normalization 
factor P may be, for example, a value obtained by dividing a 
distance from the origin to a most remote viewpoint that a 

15 user can take in the surrounding-image-display coordinate 

system by a distance from an object to a surrounding camera 
22 k that is farthest from the object. 

After the process of step S42, the processing proceeds 
to step S43, in which the image selecting unit 66 calculates, 

2 0 as a minimum camera distance, a minimum value D min among the 
camera distances calculated for the m-th viewpoint in step 
S42, and the processing proceeds to step S44 . In step S44, 
the image selecting unit 66 determines whether the minimum 
camera distance D min for the m-th viewpoint is not greater 

25 than (or is smaller than) a predetermined threshold value 
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TH2 . 

If it is determined in step S44 that the minimum camera 
distance D min for the m-th viewpoint is not greater than the 
predetermined threshold value TH2 , that is, if the camera 
5 viewpoint with the minimum camera distance D min is close to 
the m-th viewpoint and can therefore be considered as the m- 
th viewpoint, the processing proceeds to step S45. In step 
S45, the image selecting unit 66 selects, as a viewpoint 
image for the m-th viewpoint, an i-th frame camera image 

10 captured by a surrounding camera 22 k having a camera 

viewpoint with the minimum camera distance D min , from among 
the surrounding image data stored in the frame memory 65, 
and supplies it to the viewpoint -image converting unit 67. 
The processing then proceeds to step S49. 

15 On the other hand, if it is determined in step S44 that 

the minimum camera distance D min for the m-th viewpoint is 
greater than the predetermined threshold value TH2 , that is, 
if the camera viewpoint with the minimum camera distance D min 
is not close to the m-th viewpoint and therefore cannot be 

2 0 considered as the m-th viewpoint, the processing proceeds to 
step S47. In step S47, the image selecting unit 66 detects, 
as a second camera distance D 2 , the second smallest camera 
distance calculated for the m-th viewpoint, and selects two 
camera images from among the surrounding- image data stored 

2 5 in the frame memory 65, namely, an i-th frame camera image 
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captured by a surrounding camera 22 k having a camera 
viewpoint with the minimum camera distance D min and an i-th 
frame camera image captured by a surrounding camera 22 k 
having a camera viewpoint with the second camera distance D 2 . 
5 Furthermore, in step S47, the image selecting unit 66 
combines the i-th frame camera image captured by the 
surrounding camera 22 k having the camera viewpoint with the 
minimum camera distance D min and the i-th frame camera image 
captured by the surrounding camera 22 k having the camera 

10 viewpoint with the second camera distance D 2 by a ratio 
D 2 :D min of the second camera distance D 2 and the minimum 
camera distance D min/ thereby generating a synthesized image. 

That is, letting a c-th pixel constituting the i-th 
frame camera image captured by the surrounding camera 2 2 k 

15 having the camera viewpoint with the minimum camera distance 
D min be denoted as p c , and a c-th pixel constituting the i-th 
frame camera image captured by the surrounding camera 2 2 k 
having the camera viewpoint with the second camera distance 
D 2 be denoted as q c , the image selecting unit 66 generates a 

2 0 synthesized pixel by combining the pixels p c and q c according 
to a formula (D 2 x p c + D min x q c ) / (D 2 + D min ) , and generates a 
synthesized image composed of such synthesized pixels. 

Although a synthesized image is generated using two 
camera images in the example given above, alternatively, a 

2 5 synthesized image may be generated using three or more 
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camera images. 

The processing then proceeds to step S48, in which the 
image selecting unit 66 selects the synthesized image 
generated in step S47 as a viewpoint image for the m-th 
5 viewpoint, and supplies it to the viewpoint -image converting 
unit 67. The processing then proceeds to step S49. 

In step S49, the viewpoint image for the m-th viewpoint 
is converted into a display image. More specifically, in 
step S49, the viewpoint -direct ion calculating unit 64 

10 calculates a viewpoint direction (0, <|>) of the m-th 

viewpoint stored in the viewpoint storing unit 63, and 
supplies it to the viewpoint -image converting unit 67. The 
viewpoint -image converting unit 67, based on the viewpoint 
direction (0, <|>) of the m-th viewpoint, supplied from the 

15 viewpoint -direction calculating unit 64, calculates a 
position of a display image where each pixel of the 
viewpoint image for the m-th viewpoint, supplied from the 
image selecting unit 66, is to be mapped, according to 
equations (8) and (9) . Then, the viewpoint -image converting 

20 unit 67 maps (writes) each pixel of the viewpoint image for 
the m-th viewpoint to a position (address) on the frame 
memory 68 corresponding to the position of the display image 
thus calculated. The processing then proceeds to step S50. 
Then, light beams corresponding to the display image 

25 written to the frame memory 68 are emitted from the inner 
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omnidirectional projector 43, whereby the inner dome screen 
'44 displays an image of the object as viewed from the m-th 
viewpoint . 

In step S50, the controller 69 determines whether the 
5 variable m equals the total number M of the viewpoints 
stored in the viewpoint storing unit 63. If it is 
determined in step S50 that the variable m does not equal M, 
the processing proceeds to step S51, in which the controller 
69 increments the variable m by 1 . The processing then 

10 returns to step S42, and the same processing is repeated. 

On the other hand, if it is determined in step S50 that 
the variable m equals M, that is, if viewpoint images for 
all the viewpoints stored in the viewpoint storing unit 63 
have been converted into display images and the display 

15 images have been written to the frame memory 68, the inner 
omnidirectional projector 43 (Fig. 8) emits light beams 
corresponding to the display images stored in the frame 
memory 68 as images of the i-th frame. Accordingly, the 
inner dome screen 44 displays bird's-eye images of the 

2 0 objects as viewed respectively from the first to M-th 

viewpoints at positions in accordance with the respective 
viewpoints. 

Then, the processing proceeds to step S52, in which the 
controller 6 9 refers to the frame memory 65 to determine 
25 whether the i-th frame is the last frame. If it is 
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determined in step S52 that the i-th frame is not the last 
frame, that is, if surrounding -image data of a frame 
subsequent to the i-th frame is stored in the frame memory 
65, the processing proceeds to step S53 . In step S53, the 
5 controller 69 increments the variable i by 1. The 

processing then returns to step S22 shown in Fig. 28, and 
the same processing is repeated. 

On the other hand, if it is determined in step S52 that 
the i-th frame is the last frame, that is, if surrounding- 

10 image data of a frame subsequent to the i-th frame is not 
stored in the frame memory 65, the surrounding- image 
processing is exited. 

As described above, in the display system 2, an 
immersion image of the i-th frame is displayed on the inner 

15 surface of the outer dome screen 42, and a bird's-eye image 
of the i-th frame is displayed on the outer surface of the 
inner dome screen 44. Accordingly, the user is allowed to 
experience a sense of immersion provided by the immersion 
image, and is also allowed to readily grasp his/her own 

20 position by the bird ! s-eye image. 

Thus, for example, if the imaging/display system shown 
in Fig. 1 is used for imaging and display of a sports 
program, a large number of surrounding cameras 22 k (Fig. 4) 
of the imaging system 1 is disposed successively so as to 

25 surround a stadium where the sports event is being held, and 
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the omnidirectional camera 21 (Fig. 3) is disposed at a 
center of the stadium so as to allow imaging. Accordingly, 
the display system is allowed to display an image that 
provides a sense of presence. 
5 Furthermore, the inner dome screen 44 displays a 

bird's-eye image in accordance with a viewpoint of a user at 
a position in accordance with the viewpoint of the user, so 
that the bird's-eye image presented to the user changes as 
the user moves his/her viewpoint. Thus, the user is allowed 

10 to "look into" an object, that is, moves his/her viewpoint 
to view an object that has been hidden as viewed from a 
certain viewpoint. Furthermore, bird's-eye images of an 
object as viewed from respective viewpoints of a plurality 
of users can be presented to the plurality of users. 

15 Furthermore, with the display system 2, for example, 

the outer dome screen 42 (Fig. 7) displays fish, animals, 
etc. that are remote while the inner dome screen 44 (Fig. 8) 
displays fish, animals, etc. that are nearby, thereby 
virtually implementing a nature observing facility such as 

2 0 an aquarium or zoo. 

Furthermore, the display system 2 allows display of an 
artificially produced image such as computer graphics 
instead of an image actually captured. That is, the display 
system 2 allows display of a computer graphic image of the 

25 cosmic space, microscopic structure of molecules, etc. In 
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that case, an interface that allows a user to intuitively 
grasp a structure that cannot usually be recognized by the 
vision is provided. 

Furthermore, the display system 2 can be used, for 
5 example, for displaying images in what is called a video 
game . 

The series of processing steps by the wide -angle -image 
processing unit 33 and the surrounding- image processing unit 
3 5 shown in Fig. 6 may be executed either by hardware or by 

10 software. When the series of processing steps is executed 
by software, programs constituting the software are 
installed, for example, on a general -purpose computer. 

Fig. 3 0 shows an example configuration of a computer on 
which the programs for executing the series of processing 

15 steps described above are to be installed, according to an 
embodiment . 

The programs can be stored in advance in a hard disc 
105 or a ROM 103, which is a recording medium included in 
the computer. 

2 0 Alternatively, the programs may be temporarily or 

permanently stored (recorded) on a removable recording 
medium 111 such as a flexible disc, a CD-ROM (compact disc 
read only memory) , an MO (magneto optical) disc, a DVD 
(digital versatile disc) , a magnetic disc, or a 

25 semiconductor memory. The removable recording medium 111 
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can be provided as so-called package software. 

Instead of installing the programs onto the computer 
from the recording medium 111 as described above, the 
programs may be transferred by wireless from a downloading 
5 site to the computer via an artificial satellite for digital 
satellite broadcasting, or transferred through wires via 
networks such as a LAN (local area network) and the Internet 
so that the programs transferred to the computer will be 
received by a communication unit 108 and can be installed 

10 onto the internal hard disc 105. 

The computer includes a CPU (central processing unit) 
102. The CPU 102 is connected to an input/output interface 
110 via a bus 101. When the CPU 102 receives input of a 
command issued by a user by operating an input unit 107 

15 including, for example, a keyboard, a mouse, and a 

microphone, the CPU 102 executes a program stored in the ROM 
(read only memory) 103 according to the command. 
Alternatively, the CPU 102 loads into a RAM (random access 
memory) and executes a program stored in the hard disc 105, 

20 a program transferred via a satellite or a network, received 
by the communication unit 108, and installed on the hard 
disc 105, or a program read from the removable recording 
medium 111 and installed on the hard disc 105. The CPU 102 
thus executes the above-described processing according to 

25 the flowcharts or processing executed with the above- 
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described arrangements shown in the block diagrams. The CPU 
102 then outputs the processing result from an output unit 
106 including, for example, an LCD (liquid crystal display) 
and a speaker, sends it from the communication unit 108, or 
5 records it on the hard disc 105 as required, for example, 
via the input/output interface 110. 

The processing steps defining the programs for allowing 
the computer to execute the various processes in this 
specification need not necessarily be executed sequentially 
10 in the time order shown in the flowcharts, and may be 
executed in parallel or individually (e.g., parallel 
processing or processing using objects). 

Furthermore, the programs may be executed either by a 
single computer or by distributed processing with a 
15 plurality of computers. Furthermore, the programs may be 
transferred to and executed by a remote computer. 

In the display system 2, an image recorded on the 
recording medium 3 (Fig. 1) can be displayed in an off-line 
manner, and also, an image transmitted via the transmission 
2 0 medium 4 can be displayed in real time. 

Furthermore, as described earlier, the inner 
omnidirectional projector 43 (Fig. 8) emits light beams 
corresponding to a display image, and the light beams are 
back-projected onto the inner dome screen 44 (Fig. 8) ; thus, 
25 when a viewpoint image is displayed, the left and right of 
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the viewpoint image is reversed. In order to avoid the 
left -right reversal of the viewpoint image displayed on the 
inner dome screen 44, for example, the left and right of a 
camera image that serves as an original for the viewpoint 
5 image must be reversed in advance. This reversal can be 

executed, for example, by the image selecting unit 66 (Fig. 
23) of the surrounding- image processing unit 35. 
Alternatively, the left-right reversal of the camera image 
may be executed optically when the camera image is captured 

10 by a surrounding camera 22 k (Fig. 4) of the imaging system 1. 
Furthermore, in this embodiment, the inner 
omnidirectional projector 43 (Fig. 8) emits light beams 
corresponding to a display image, and the light beams are 
back-projected to the inner dome screen 44 (Fig. 8) , whereby 

15 a viewpoint image is displayed. In this case, the viewpoint 
image must be generated with consideration of the shape of 
the inner dome screen 44 (Fig. 8) . More specifically, 
considering that the viewpoint image is rectangular, when 
the rectangular viewpoint image is mapped to a particular 

2 0 position of the display image, light beams corresponding to 
the display image are emitted by the inner omnidirectional 
projector 43, whereby the viewpoint image is displayed on 
the inner dome screen 44, since the rectangular viewpoint 
image is projected onto the spherical surface of the inner 

2 5 dome screen 44 having a shape of a hemisphere, the viewpoint 
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image displayed on the inner dome screen 44 is distorted. 
Thus, in order that the inner dome screen 44 display the 
viewpoint image in a rectangular shape without distortion, 
the viewpoint -image converting unit 67 (Fig. 23) must 
5 convert the viewpoint image and map it to a display image 
based on the shape of the inner dome screen 44. 

In this embodiment, however, the radius of the inner 
dome screen 44 (Fig. 8) is large to a certain extent, and 
the range where the viewpoint image is displayed can be 

10 considered as planar. In this case, the conversion and 

mapping of the viewpoint image to a display image based on 
the shape of the inner dome screen 44 is optional. 

Furthermore, although only one surrounding imaging unit 
12 (Fig. 2) is disposed in the imaging system 1 in this 

15 embodiment, a plurality of surrounding imaging units may be 
provided. In that case, the arrangement may be such that 
the plurality of surrounding imaging units captures images 
of a plurality of objects that are spatially or temporally 
separated, and that the display system 2 processes an image 

20 obtained by superimposing the images of the plurality of 
objects as a camera image. 

Furthermore, the wide -angle -image processing unit 33, 
the surrounding- image processing unit 35, and the viewpoint 
detecting unit 37 shown in Fig. 6 may be provided in the 

25 imaging system 1 (Fig. 2) instead of the display system 1. 
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The present patent specification contains subject 
matter related to that disclosed in Japanese priority 
document, application number P2002 -291819 , filed in the 
Japanese patent office on October 4, 2002, the entire 
5 contents of which being incorporated herein by reference. 

For convenience, the following reference numerals are 
used throughout the figures to represent the various 
components employed in one or more embodiments according to 
the present invention. 1: imaging system; 2: display 

10 system; 3: recording medium; 4: transmission medium; 11: 
wide-angle imaging unit; 12: surrounding imaging unit; 13: 
multiplexer; 14: recording/sending unit; 21: 
omnidirectional camera; 22 k : surrounding cameras; 31: 
playback/receiving unit; 32: demultiplexer; 33: wide- 

15 angle-image processing unit; 34: wide-angle-image display 
unit; 35: surrounding- image processing unit; 36: 
surrounding- image display unit; 37: viewpoint detecting 
unit; 41: outer omnidirectional projector; 42: outer dome 
screen; 43: inner omnidirectional projector; 44: inner 

20 dome screen; 51: frame memory; 52: wide -angle -image 

converting unit; 53: image-angle correcting unit; 54: 
latitude-longitude-image converting unit; 55: frame memory; 
56: controller; 61: viewpoint buffer; 62: viewpoint 
processing unit; 63: viewpoint storing unit; 64: 

25 viewpoint-direction calculating unit; 65: frame memory; 
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66: image selecting unit; 67: viewpoint - image converting 
unit; 68: frame memory; 69: controller; 70: display-image 
generating unit; 101: bus; 102: CPU; 103: ROM; 104: RAM; 
105: hard disc; 106: output unit; 107: input unit; 108: 
communication unit; 109: drive; 110: input /output 
interface; 111: removable recording medium 



